On Rater Agreement and Rater Training

On Rater Agreement and Rater Training

This paper first analyzed two studies on rater factors and rating criteria to raise the problem of rater agreement. After that the author reveals the causes of discrepencies in rating administration by discussing rater variability and rater bias. The author argues that rater bias can not be eliminated completely, we can only reduce the error to a cetain degree by training raters. The study on r...

Rater agreement in emerging psychosis

Inter-rater Agreement on Sentence Formality

Formality is one of the most important dimensions of writing style variation. In this study we conducted an inter-rater reliability experiment for assessing sentence formality on a five-point Likert scale, and obtained good agreement results as well as different rating distributions for different sentence categories. We also performed a difficulty analysis to identify the bottlenecks of our rat...

Rater Effects as a Function of Rater Training Context

This study examined the influence of rater training and scoring context on the manifestation of rater effects in a group of trained raters. 120 raters participated in the study and experienced one of three training/scoring contexts: (a) online training in a distributed scoring context, (b) online training in a regional scoring context, and (c) stand-up training in a regional context. After trai...

The impact of training on rater variability

In the five years, 1993 to 1998, total Commonwealth Government spending on education fell from 4.9 to 4.4 per cent of Gross Domestic Product. Australian universities have responded to this changed funding environment though the increased use of casual teaching staff. The aim of this study was to develop, implement and evaluate a short, cost-effective training package designed to improve the rat...